Accuracy of Simulated Flat, Combinatorial, and Penalized Prediction Markets

There is preliminary empirical evidence that a combinatorial prediction market (CPM) can outperform a flat prediction market (FPM), but the conditions under which gains are realized are unclear. In a study with simulated agents, we found that CPMs fare better than FPMs when few agents have specialized knowledge and events are strongly related. By allowing conditional forecasts, a CPM is over 50% more accurate than an FPM in the FPM’s worst-case scenario. We also consider a novel prediction market that directly penalizes probabilistic incoherence. We consider accuracy in crowdsourced forecasts. Recent work has shown long-run accuracy well above a simple average [Mellers et al. 2014]. Two promising nondemocratic approaches addressed in this study are prediction markets and coherence weighting. In a prediction market, forecasters buy and sell contracts on verifiable future events. Market prices for contracts are interpreted as the probability of the event. When the event resolves, forecasters gain (lose) assets if they bought contracts representing the outcome that occurred (did not occur). Therefore accurate forecasters tend to increase their influence over time, while inaccurate forecasters tend to decrease their influence over time. More information can be found in [Arrow et al. 2008; Pennock et al. 2001]. Another nondemocratic approach weights by coherence under the assumption that more coherent forecasts are more accurate. Coherence is available at forecast time, and provides assessment even when estimating objective probabilities is difficult [Lindley et al. 1979]. Recent studies have shown that weighting forecasters by the degree to which their forecasts are probabilistically coherent can also substantially improve upon simple averages, at least in knowledge tasks [Karvetski et al. 2013; Olson and Karvetski 2013; Tsai and Kirlik 2012; Wang et al. 2011]. De Finetti showed that under proper scoring rules, any incoherent set of forecasts can be replaced by a coherent set that has a better score for every possible outcome [de Finetti 1937; 1981]. Therefore, if events in a market are related, we might expect that markets which disallow or penalize incoherence will outperform those which allow incoherence.